Unsupervised Detection of APT C&C Channels using Web Request Graphs

HTTP is the main protocol used by attackers to establish a command and control (C&C) channel to infected hosts in a network. Identifying such C&C channels in network traffic is however a challenge because of the large volume and complex structure of benign HTTP requests emerging from regular user browsing activities. A common approach to C&C channel detection has been to use supervised learning techniques which are trained on old malware samples. However, these techniques require large training datasets which are generally not available in the case of advanced persistent threats (APT); APT malware are often custom-built and used against selected targets only, making it difficult to collect malware artifacts for supervised machine learning and thus rendering supervised approaches ineffective at detecting APT traffic. In this paper, we present a novel and highly effective unsupervised approach to detect C&C channels in Web traffic. Our key observation is that APT malware typically follow a specific communication pattern that is different from regular Web browsing. Therefore, by reconstructing the dependencies between Web requests, that is the Web request graphs, and filtering away the nodes pertaining to regular Web browsing, we can identify malware requests without training a malware model. We evaluated our approach on real Web traces and show that it can detect the C&C requests of nine APTs with a true positive rate of 99.5100% and a true negative rate of 99.5-99.7%. These APTs had been used against several hundred organizations for years without being detected.